The main idea behind FI-STAR was that implementing some of the common requirements of e-health application in the form of FI-GE and FI-SE in the FIWARE platform can foster the development of e-health applications, and the FI technology platform can create a sustainable ecosystem from those applications. Also, it was considered that the cloud architecture of FI can be designed to accommodate software to data approach, where the applications migrate to data centers, eliminating the need for data to be transferred to other locations. This special architecture is supposed to address many of the privacy and regulatory issues in health information systems (FISTAR Consorium, 2014) that prevent this sector from enjoying the same cloud-based technological advancement as other sectors.

Each of the FI-STAR use-case or trial was located in a different site (Munich had two trials) Figure 2.1. Tromso case developed a mobile-based health information system for providing tele-health services for diabetes and their physicians, in terms of demand-based sharing and visualization of data. Leeds developed a barcode application for reverse tracking pharmaceutical products in the supply chain, making sure about authenticities. Krakow developed a tele-medicine solution both for rehabilitation of cancer patients and preparation for surgery. Bilbao in Basque country in Spain developed an interactive system to support patients with bipolar and other mental disorder to have access to services by health professionals. Bucharest developed a solution for monitoring and delivering health care services to patients with heart failure problem, especially in the rehabilitation phase. Munich, in the first case, developed a solution to track consumables in the operation room, making sure none has remained in the patients body during surgery. Munich, in its second case, developed a solution to facilitate the transportation of people with mobility problem to health care centers by advising routes of public transport services. This solution was not in the initial set of use-cases to evaluate, but we include this in our final evaluations using the questionnaire developed based on other cases. Bologna in Italy developed a solution for sharing information between patients and health professionals, especially in COPD ^[1] case.

In health technology assessment studies, special attention was paid to the historical rationals and the scope of topic (Banta & Jonsson, 2009) (Banta, 2003) (Draborg, Gyrd-Hansen, Bo Poulsen, & Horder, 2005) (Luce et al., 2010) (Nielsen, Funch, & Kristensen, 2011) (Lim et al., 2009); models, methods, aspects being considered, and reporting (Douma, Karsenberg, Hummel, Bueno-de-Mesquita, & van Harten, 2007) (Goodman & Ahn, 1999) (Lampe et al., 2009) (Kristensen et al., 2009) (Kristensen, 2009) (Pasternack et al., 2014) (Ramacciati, 2013) (Shekelle, Morton, & Keeler, 2006) (Turner et al., 2009).

In evaluation of health information systems, special attention was paid to prospect (Haux, 2010) (Haux, 2006) (Ückert et al., 2014) (Talmon & Hasman, 2002); challenges (Ammenwerth, Gräber, Herrmann, Bürkle, & König, 2003) (Berg, 2001) (Bernstam et al., 2010) (Greenhalgh & Russell, 2010) (Greenhalgh et al., 2010) (Gremy, Fessler, & Bonnin, 1999) (Haux, 2010)(Heeks, 2006) (Littlejohns, Wyatt, & Garvican, 2003) (Shiell, Hawe, & Gold, 2008) (Paley & Eva, 2011) (Topolski, 2009); strategies for better evaluations (Ammenwerth et al., 2004) (Campbell et al., 2000) (Lilford, Foster, & Pringle, 2009) ; models, methods, and reporting (Holden & Karsh, 2010) (Hu, Chau, Liu Sheng, & Tam, 1999) (Hyppönen, Ammenwerth, Nohr, Faxvaag, & Walldius, 2012) (Kidholm et al., 2012) (Wu et al., 2006) (Wyatt & Wyatt, 2003) (Yen & Bakken, 2012) (Williams, May, Mair, Mort, & Gask, 2003) (Yusof, Papazafeiropoulou, Paul, & Stergioulas, 2008) (Brender et al., 2013) (Talmon et al., 2009) (Nykänen et al., 2011) (Ekeland, Bowes, & Flottorp, 2012). In ontology engineering topic, special attention was paid to application of ontologies specially in health information systems (Dong & Hussain, 2011) (Gurupur, Sakoglu, Jain, & Tanik, 2014) (Noy, 2004) (Noy & McGuinness, 2001) (Ovaska, Evesti, Henttonen, Palviainen, & Aho, 2010) (Pinto, Gómez-Pérez, & Martins, 1999) (Long, 2001) (Schugerl, Rilling, Witte, & Charland, 2009) (Wang, Zhang, Gu, & Pung, 2004) (Fernández-López & Gómez-Pérez, 2002) (Hitzler, Krötzsch, Parsia, Patel-Schneider, & Rudolph, 2012).

Based on the literature survey, the UVON method was first developed in the study 1 and then extended in the study 2. The method was applied in FI-STAR and its seven e-Health application use-cases, for summative evaluation of the individual and whole e-health applications. The UVON application resulted in an ontology of 400 nodes (of 2 digits of significant) in an ontology hierarchical structure, where the top level 10 quality aspects and their children were considered for formation of the questionnaires (refer to Questionnaires). The questionnaires, made in two forms for patients and health professionals, were delivered to all use-cases (including an 8th use-case that was added later to the project) that resulted in 87 answers from the patients and 30 from the professionals.

Due to practicalities of the FI-STAR project, the use-cases have been asked to participate in the data collection phase. This can be prone —as it can be usual in similar survey-based approaches— to human errors, conflict of interests, lower degrees of accuracy, and all other uncertainties associated with this type of measurement.

This vast landscape of health technology definition increases the complexity or cost and decreases the efficiency or effectiveness of any health technology related activity, such as evaluation, that wants to make an inclusive, comprehensive, and uniform approach towards the health technology topic.

Conquering the vast health technology landscape through dividing into sub-disciplines is a strategy that we already do, but it can also be challenging. Our diverse cognitive sense and the historical reasons influence how we categorize health technologies. In the above health technology definition by INAHTA, we can recognize implied divisions by pharmaceuticals, devices, procedures and organizational systems categories, while, the boundaries between these divisions is not very clear. For example, a procedure can be a surgical procedure, a procedure within operation team, or an organizational procedure. The first and the second type of procedures can be grouped into one while we can do the same for the second and third procedures. At the same time, the first and the third procedures can be considered far enough from each other to belong to different sub-groups of the health technology. This example can be extended to pharmaceuticals and devices subgroups. For example, stomach-resident devices and ingestible electronics (Zhang et al., 2015) are moving toward blurring the difference between devices and pharmaceuticals.

This early definition of information technology departs clearly from wide possible definitions that could include even the Sumerian tablets in their defined scope of the information technology. A definition such as the above might be too limited, or might need revisions along the technological advancements, but it can be more practical than definitions that are too much inclusive. Pragmatism in bounding the scope of information technologies enables us to introduce practical evaluation methods for those technologies; still it might ignore the more accurate nature of the phenomenon. The balance between pragmatism and comprehensiveness perspectives can be a challenge for the evaluation.

Probably a more clear account of what we mean by information technology can be addressed by the term informatics and respectively health informatics. The term informatics is coined by Karl Steinbuch in 1957 (Steinbuch, 1957) in his book “Informatik: Automatische Informationsverarbeitung”, which translates to “Informatics: Automatic Information Processing” and shows a separation from other forms of information processing by characterizing it as being automatic. By this definition —and if we ignore some early analog computing devices such abacus and astrolabe which were also very use-case-specific for accounting and astronomy— the new automated computing devices are all those electronic devices, usually based on transistor technology, that automate information processing. Though, this perspective lacks the communication dimension, which nowadays we emphasize more by using the term ICT instead of IT; but we can extend our view —but probably not the informatics term— when talking about the evaluation of health information technology to include both the automated processing and communication.

The terms health information technology and health information system might be used interchangeably in some contexts, but each of these terms has some connotations that we should be clear about them. Evaluation of a specific health information technology instance is usually about evaluating that technology in different applications and cases, which gives a wider perspective of evaluation in contrast to the evaluation of a health information system that is about an enumerated set of health information system implementations. From the other hand, a health information system can recruit more than one health information technologies, where all those heterogeneous technologies have taken part in the whole of that health information system.

The literature of health technology assessment health technology assessment emphasizes on its role in improving policy makings related to health technology (Banta, 2009) and considers lots of economical considerations that implies the health technology in health technology assessment is less concerned about specific instances. From the other side, relying more on adoption and acceptance of technology (Ammenwerth, Iller, & Mahler, 2006; Holden & Karsh, 2010) shows health information system evaluation literature is more concerned about specific implementations that consists of technologies and human agents combined in the form of a unique system. It can be imagined that this separation between domains of concern is not very crisp and clear in all cases or studies, still a bit of more clarification can explain more about the intentions of an evaluation or a related study.

The holistic and functional perspective on health information systems also implies that a health information system can maintain its core characteristics and functionality while the underlying technologies can be changed totally. This is a drastic departure from the health technology and its volatile character. In the holistic and functional view, the technological changes do not change our perspective and approach to evaluation of health information systems and the evaluation aspects maintain to live much longer than the life of the underlying technologies.

The criteria for success in technology, in general, and health technology, in specific, have been discussed in different studies. These criteria go beyond the intended functionalities of system to include qualities they expose (Holden & Karsh, 2010; Berg, 2001).

Models such TAM, TAM2, UTAUT put the acceptance as the cornerstone of success in a technology implementation (Hu, Chau, Liu Sheng, & Tam, 1999)(Venkatesh, Morris, Davis, & Davis, 2003), where acceptance can be detailed more as usage when it is voluntary or keep it as the overall user acceptance when then usage is mandatory (Goodhue & Thompson, 1995; Ammenwerth, Iller, & Mahler, 2006). TAM and TAM2 put behavioral intention to use (acceptance) at the center and then expand it perceived usefulness and perceived ease of use determinants (Davis, 1989; Holden & Karsh, 2010). Almost similar to TAM and TAM2, UTAUT considers performance expectancy, effort expectancy, social influence,and facilitating conditions as the determinants of acceptance. The TTF puts the fit between the task and technology as the major indicator of success (Goodhue & Thompson, 1995). The FITT model puts the interaction between the user and task in the TAM and TTF combination by creating a triangle of fitting relations between task, technology, and individual (Ammenwerth, Iller, & Mahler, 2006).

The answer to what to evaluates need some insights about the scope and nature of evaluation, which some of them is discussed in the rest of this chapter.

While the responses mentioned above provide unified frameworks for evaluating different health technology cases in a universal form, but they are static frameworks with no mechanism for accommodating unforeseen, time-variant, or context-specific evaluation aspects. Being static and unable to accommodate new aspects, and at the same time trying to be universal, in contrast to being case specific, can weaken the relevance relation between those frameworks and a case, hence making it challenging to apply the framework.

Clinical (or epidemiological) episode can be of decisive importance for impact evaluation of any health care intervention. The evaluation of clinical episode is usually dominated by clinical trial methodology (Williams, May, Mair, Mort, & Gask, 2003); but for health care information system it is subjected to some challenges and limitations, such as those in application of randomized controlled trials or identifying objective parameters to evaluate (Shcherbatykh, Holbrook, Thabane, & Dolovich, 2008; Bürkle, Ammenwerth, Prokosch, & Dudeck, 2001). From the other end, evaluation of a health care information system just as a standalone system, i.e. at the very early episode of impact propagation journey, can barely be considered a matter of health information technology topics. An isolated or technical evaluation of health care information system is probably a matter of concern for other technology disciplines, such as software engineering. Between these initial and ending episodes of impact propagation, there exist one or more episodes where the health care information system can be evaluated by its impact on the surrounding health setting, processes, or knowledge. As defined by WHO the health setting is "the place or social context in which people engage in daily activities in which environmental, organizational, and personal factors interact to affect health and wellbeing" (World Health Organization, 1998). This definition can be detailed or be extended by taking into consideration the processes and embodiments of knowledge within that space or context beyond just the entities. Many evaluation frameworks focus on addressing the evaluation of this range of impact propagation episodes and their corresponding health settings.

In the spatial dimension, references to evaluation aspects indicate, explicitly or implicitly, the context, the environment, or the actors in the space for which the evaluation should be or can be performed. For example, the organizational aspect mentioned in MAST framework (Kidholm et al., 2012) suggests the organization scope and its relevant members as the scope of the evaluation.

In this model, an ecosystem is formed around a set of health care information systems that are targets of evaluation. The health care information systems and the digital environment where they reside, and probably interact with each other, is called the digital actors zone (the \(D\) zone). The community of other digital software or hardware entities, more likely working as infrastructures serving digital information and communication technology services to the digital actors and also the user actors, is recognized as another zone, called digital infrastructure zone or simply the infrastructure zone (the \(I\) zone). The infrastructure zone has more blurred boundaries due to its definition. The users community and their interaction with each other, related to those health care information systems, is called user actors zone (the \(U\) zone). The social and the physical environment that hosts these users is recognized as socio-physical zone (the \(S\) zone). This zone also lacks crisp boundaries due to its definition.

The model emphases that each of these four zones is neighboring the other three ones. These adjacencies create six connection points between the zones. Each of these six connection points represents the flow of value, information, or other exchangeable objects between the two zones. We can consider that each connection point is made of two unidirectional channels, each lets the flow of value, information, etc. in the opposite direction of the other. In this sense, we have twelve connection channels, through which a zone, or one or more of its members, receives values or information from another zone, or one or more of its members. Each of these channels can be shown by a notation like \(X \longrightarrow Y\), where \(X\) and \(Y\) can be any of \(D\), \(U\), \(I\), \(S\) zone symbols.

We should also pay attention to two types of interventions, the one that wants to return the situation to a predefined baseline situation and the one that want to improve the current situation beyond the previous history. In this sense, evaluations at clinical level can usually be categorized in the first group where it is assumed that there is a normal, albeit adjusted, healthy situation for an individual (or a population), and the intervention should just revert the physiological or psychological situation to that state. However, the impact on health setting might be of both improvement or reverting nature. Evaluation time scope for the reverting case is related to the baseline situation, the probability of fully reverting, and the speed of reverting. While the evaluation of improvement case is related to the stability of changes and impacts.

Things get more complicated if we assume there is no permanent impact; hence each impact is subjected to some attrition or deformation rate. Even more, the goals that the impacts are supposed to fulfill can erode along the time (Greenhalgh & Russell, 2010). It is not a guaranteed grace to find a period during the life course of a heterogeneous health setting including the under study health care information system, where all the impacts have reached the state of maturity and stability but none has reached attrition.

Another challenge with suggesting a temporal framework for evaluation is the context of evaluation itself. The evaluations that are part of policy making, project assessment, investment performance evaluation, or other time-bounded activities should comply with practical considerations of the whole activity rather than just focusing on the best time for evaluation. The value of long scopes of time in evaluation can be challenged if it is anticipated that new interventions, of totally new characteristics, might replace the current intervention. In this regard, health technology assessment literature has not much incorporated the dynamic nature of technology in the assessment (Douma, Karsenberg, Hummel, Bueno-de-Mesquita, & van Harten, 2007).

In the presence of insight about the networked and complex nature of health care socio-technical environment (Shiell, Hawe, & Gold, 2008), beside both the intended and unintended direct impacts, the emergence type of impacts that are not associated directly with individual health care information systems are also matters of concern. Emergence behaviors are those behaviors, usually unintended, that appear in a complex system as a result of individual members' activity and interactions, whereas those behaviors are not the properties of any of those individuals separately and cannot be evaluated by observing at the scope of an individual (Halley & Winkler, 2008). The intervention of a health care information system within a complex health setting, let’s say ecosystem, can contribute to the formation of some emergent behaviors and emergent impacts.

Impacts of an intervention, in a counterfactual manner, are recognized by differences between the condition of presence and absence of the intervention (Ferraro, 2009). In this sense, evaluating the impact of an intervention also encompasses those effects that are side-effect, unintended or are part of the intervention structure and embodiment. For example, the learning curve of a new health care information system and the time allocation needed to address that is an unintended impact, which is usually tried to be minimized as much as possible. Sometimes it can be tricky to recognize if a quality refers to an intended value or an unintended effect. For example, a health care information system might increase efficiency in a health setting by reducing the number of tasks hence the efficiency is the intended impact of that health care information system; but at the same time, another health care information system, such as a medical image processing application, can also be efficient, in that sense that it does not take too much time for creating its intended results. In the first case, the health care information system contributes to the total efficiency as it is intended, whereas in the second case it avoids contributing to inefficiencies by being efficient itself. Here in the second case, efficiency is none primary or an unintended impact. With this insight, evaluating the impacts of a health care information system, in episodes before the clinical episode, involves evaluating the main intended impacts, side effects, or by-products of the intervention.

Although it is not always clearly mentioned, the each ontology is essentially a network graph, with a finite number of node elements, connected through a finite number of relations. Semantic networks are also essentially the same concept, with historically different roots; of course, different implementations of ontology or semantic network notations might be different in features. In some of the minimal-form ontologies, such as a list of concepts, the relations are not mentioned explicitly, but there are always at least one implied relation that considers all the elements in a set to be children of a parent concept. This parent concept can be the name of the set, or at least it can be the thing concept that is a parent to any other concept. Many of other ontologies represent at least the hierarchical relations between the concepts or objects, specifying which concept or object is the parent, super-class, or the more abstract form of some other concepts or objects; or vice versa, which is the child, sub-class, or more specific concept to which.

Ontologies can be represented in different notations. Verbal, tabular, or graph-based notations can have the same functionality in their representations. Many of the ontologies are in plain text format but representing to a computing-friendly format can help maintenance, development, integration, and data-mining of those ontologies. Each discipline has its favorite notation for their related ontologies. For example, the OBO format is a common format in bio-medical science, while OWL as a generic ontology notation.

Even with one case, and ontology empowers us to infer more about the nature of the case. An ontology can capture many aspects of a system in its network and feed them to inference and data mining engines. Aggregation, unification of concepts, and integration between few cases are amongst other reasons that ontology construction can be applied to a class of cases with few members.

Ontological representation of a system exposes it to automated or semi-automated evaluation algorithms. These algorithm create, expand, and traverse the ontology in the sake of inferring new knowledge. A hierarchical ontology embeds the essence of unification and aggregation in its hierarchies. This essence of unification and aggregation is used in study 1 and 2 to represent a semi-automatic method for flexibly aggregating the heterogenous evaluation aspects.

The UVON method was discussed in its initial form in this study, where the method creates an ontology, based on the quality attributes elicited from the subject system(s) or extracted from a relevant evaluation model. The study applies the UVON method to the FI-STAR project and its 7 use-cases for a summative evaluation. The result of the UVON application in the FI-STAR project is a set of 10 quality attributes that are required to be evaluated across all use-cases. The quality attributes and their sub-qualities are reflected in the questionnaire in Questionnaires.

In this study, the UVON method was extended through enhancing the ontology structure by adding weighting and threshold values to the connection edges, and evaluation value to the deepest-possible nodes when it makes sense for a specific design. In this sense, each quality attribute —except the leaf nodes— is considered to be an aggregation of its child quality attributes and each child quality attribute has a share in creating the value in its parent. The shares are specified by the weighting values; also the canceling impact on the parent quality is specified by the threshold values. The deepest-possible nodes get their values by evaluating how they are satisfied by a specific design or design option. The rest of the nodes, above the nodes with initial values, get their values by summing their child values or being canceled by children that have not met the threshold.

This extended UVON method enables prioritizing between quality attributes, i.e. recognizing which one creates higher values in the upper hand nodes, and also ensures the inclusion of the critical quality aspects during the design phase. Hence, the method creates a mechanism for formative evaluation in the design phase, regarding selecting between options in a single design or comparing design alternatives with each other.

From the other side, evaluation and diagnosis are reverse forms of each other. While in diagnosis we try to find out about the causes by back casting from symptoms, in evaluation we try to find out the symptoms —let’s say quality aspects— that a known cause —let’s say a technology intervention— has created or going to create in the future.

The high degree of homogeneity between diagnosis and the ICTs main functionality and the reverse symmetry of evaluation with diagnosis creates an interesting triple overlapping zone. Insights from each of these three disciplines may create new knowledge in the other one.

This paper is under review for publication in a journal and still cannot be publicly distributed.

Requirements for a product or service can have two affirmative and negative facets. In one facet, the fulfillment of the requirement would increase the total value that the product or service delivers. In another facet, inability to fulfill a requirement or insufficiency in exposing a quality, invalidates the design and fails the product or service totally even if it excels in other aspects. For example, an electrical device in health care that scores high in price performance would be considered a failure if it could not comply with minimum required safety standards. At the same time, while electrical safety higher than required might contribute to the value the product delivers, but even lower —but enough— levels of safety does not invalidate the overall qualities.

Health care products and services are governed by extensive regulations and guidelines that are strictly needed to be observed (Johnson, 2012). These requirements are expressed and documented in a variety of documents, making it a challenge to extract and apply them via design in a consistent way.

The aggregated requirement documents of those seven trials exposed the challenges in such situations. The requirement from each of the trials were expressed in different wording, scale, perspective, and abstraction; while the nature of many of those trials were similar to each other or some of the functionalities were expected to be the same. It was anticipated that a unifying structure can be beneficial for both design and evaluation phases.

Ontologies have been used for a long time in health care to standardize, hence facilitate, the communication about health situation and health care interventions. While those ontologies are defined in global level application, but they can inspire using ontologies in small case-specific scales. Therefore, a pool of requirements were created from harvesting the requirement documents which was then structured as an ontology. There were several cases of ambiguity in constructing the ontology or interpreting the resulted ontology which led to priority and threshold mechanisms described late in this paper. In practice, the ontology is used only for the evaluation process in FI-STAR project, but its successful construction, ease of application in the evaluation phase, and its observed characteristics all together signal the feasibility of the method, with modifications specified in this paper, for design phase.

The method is elaborated with the requirements of health care systems design in mind, but it is not limited to that and can be used in other similar design contexts. A European project in health care information systems, called FI-STAR, has been the practical context of developing this method. The method tries to find a unification mechanism for the requirements communicated by the actors in a health care environment by using ontology structures.

Ontology construction is a core to the method, both for the purpose of unification and for implementing prioritizing and threshold mechanisms. Ontologies are networked representation of shared knowledge about a domain, where the concepts of that domain are represented as nodes of the network, connected to each other through a limited defined set of relations (Noy & McGuinness, 2001). Usually the concepts are represented in noun form and the relations in verb form. In this sense, the heterogeneous set of requirements in a complex health care setting can be captured within an ontology, which makes is computable for ontology related algorithms. If the relations in a requirement ontology be restricted to hierarchical relations, such as parent to child, superclass to subclass, generic to specific, or superset to subset then still the ontology can capture non-functional requirements as well as those requirements that can be expressed in form of X is of type Y form. Hierarchical relations, excluding the equity relation, impose specific forms to networks. They work only in one way, in this sense that one concept cannot be a parent, superclass, generic form, superset to itself. An extension of the above observation is that the hierarchical relations cannot chain as loop, hence any travel between nodes would be in one direction never reaching back the same node. This form of structure, i.e. direct with no loop, is called acyclic directed graph. The tree form in graphs, is a subset of acyclic direct form. If there is a single node with no parent node then it is called rooted tree. As its name implies, all nodes in a rooted tree, except the one that is called root, have only one parent. If an acyclic graph is not in tree form, then we can convert it to a tree form by making multiple instances of those nodes that have more than one parents. In a tree, all nodes get unified, level by level, through their parent node, where ultimately they get all unified in the root node. Traditionally, the root nodes in tree style ontologies are labeled thing, as every concept is ultimately a thing.

Tree style ontologies of requirements expose interesting computational characteristics. Being able to traverse between any two nodes, being able to assign each node to a level of the tree, and being able to assign values to each node and connection are some of those characteristics that play role in our method.

The method enhances the constructed ontology by assigning weighting and threshold values to the connections and quality fulfillment degree to the nodes. The enhanced ontology facilitates prioritizing the requirements in a design and detecting, hence rejecting, those designs that do not fulfill a minimum of a critical requirement. The method can also be used to communicate a set of regulations and guidelines in an integrated and clear way, by communicating a partial pre-built ontology. Any constructed ontology from a design case can be an input for other related knowledge elicitation and inference algorithms.